Predicting Query Performance for User-based Search Tasks

Query performance prediction aims to determine in advance whether a user’s search request will return a useful answer set. The success of such prediction attempts are currently evaluated by calculating the correlation between the predicted performance and standard information retrieval metrics of system performance such as average precision. However, recent work suggests that there is little relationship between average precision and the performance of users when carrying out search tasks. Direct measures of user performance offer another way of evaluating the effectiveness of search systems; this is of particular importance in the framework of query prediction, since one of the goals of prediction is to warn users when search results are likely to be poor. We therefore investigate the relationship between current prediction techniques and user-based performance measures. Our preliminary results show that the performance of the predictors differs strongly when using system-based compared to user-based performance measures: predictors that are significantly correlated with one measurement are often not correlated with the other. In general, the predictors are more correlated with average precision rather than with user performance.